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ABSTRACT . ; . ■ . f 

Prime indicators for realistic short .term/long "l^erm 
project goals .are budgets add/ timetables. Concrets,' identifiable 
objects are useful' in separating 'eloquent rhetoric /from actual 
promises.- Simi an external evaluator should be able to separate * 

proposals with intentional misrepresentation of funding and goals 
from those~ which^need further organization. CXnce a project begins, 
tlfe^eyaluator shcfald know whetfi*er^Ch§ data teing collated and 
analyzed will be used 'for internal public' consumption, external -' 
public relations, or both. This* may depend on whether the* eyalu^tors* 
primary allegiance is to th* funding agency.!-or to the project, in- any 
evaluation traditional staff' roles .and lines of authority m should be 
recognized and ^better ^communication facilitated Technical expertise "* 
and the politi-tral realities of a system should be reconciled. * - 

(bjg) . . * " 

f ' ■ * * a 
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& - John J/V. Wick 



ABOUT THIS REPORT ' . 

- /Thft article gives some very practical advice to 
.Jhe external evaluator on potential pitfalls in 
program evaluation.. Many factors relating to the 
success or failure of a. project can be found in }he 
proposal itself. Schedule, k>udge{, lines of author- 
ity and staff commitments^ all have important 
'implications*. Examine what the prop9sal pro- 
rrtised to do. Peterrpine^m advance wJiaj type of 
report is yv.afited-on£ of critical evaluation for 
internal consumption or an external one for* 
pyfrjic relatioHs;bri)oth. ^ " : 

Dr. John w 'W. Wicls, Associate Professor of 
Education at. Northwestern University, from his 
^own experience la's an evaluator lists five basic 
Characteristics important to a good evaluator: 1} 
I gather sufficient baseline * data, 2) 'Collect data 
^continuously, ^) provide low cosf feedback in 
/ understandable,fbrcnat, 4') use trend analysis, and 
5) sample where possible. ^ 

C)r. Wick has'beeo-active professionallyin areas 
generally, concerned^ with educational , measure- 
ment;* evaluation,, testing and. statistics. He^gjthe 
.author and co. author 6t hurpefous articles and 
bddks'ih his fjeldv ' - > ; * . 

* ' *•'": * y- - % cjf 
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This article is directed toward an external evaluator. ' : 
This could be> someone operating in th£ common 
situation wherein competitive bids are required for the 
external evaluation of a project. But ''external" could 
also apply to a line or staff person* within an agency, * 
serving as? a monitor for projects operating in^another 
area of the bureaucracy, or to « stajf persog in a 
funding agency monitoring outside projects. An "ex- 
ternal" 6valuator is one who wishes tcr evaluate a 
project for some reason, and who is not directly 
involved with the operation of the project.. This 'does ' 
not mean that the suggestions and comments included 
hefeiri could not be used by people internal to Yhe • 
project. They are simply not the target audience.! am 
attempting toaddress. 

Short Term/Long Termr-The Impact of the Project 

The Assistant, Superintendent of an elementary 
school district noticed two interesting things: Frrst, 
some of the students ^ were not ' making "progress ^t 
learnipg to read as fast -as he 'hoped they would; and 
second, some of the teachers weren't particularly good 
at diagnosing-fche specif ip problems these children were \ 
having. Being a .good grantsman, he wrote a proposal to 
an agency and subsequently received funding* for a 
project which was designed fo solve both problems. 

Soon a ^project administrator was hir^d; testing 
people were broyght in; and a reading "specialist was* 
assigned to each ten teachers. The reading and testing % 
specialists were to help the teachers diagnose student - 
problems. The Assistant Superintendent "was happy; 
the school board and townspeople, were impressed; the 
salaries of £ number of people were augmented; and all 
seerined right in-the world. ./ ' 

Then thejnoney ran out. *• 

The administrators" .pnoved , on to other "soft 
money"; the consultants too searched for other pro- 
jects; the teachers started misdiagnosing or ignoring 



'Much of this article is based on Chapter 12 of EDUCATIONAL * 
MEASUREMENT: Where are we going and How will we know when we ' 
get there? (Columbus, Ohio. Charles'Merrill Publishers, 1973). * 



'problems again; and. the old reading problem made a 
dramatic reappearance. ^ 

The project was a failure. Or was it? ^ 1 

Tha| depends on whether you're talking about 
short-term benefits or long term* changes. A good 
evaluate* rpust keep these two separate. • a 

In the s'hort, term, a lot of students obviously 
benefited fcom the program. The teachers and district 

-personnel probably learned from the procedures. Some 
interest,; excitement, esprit de corps, ^nd activity werfe 
generated in the district. And the program adminis- 
trators,' staff, and consultants d^l alj right, too, * '* < 
In the long run^he procedures died with the project 
funding, but maybe some positive things-did occur. 
(Vlaybe tbe funding agency learned that such an 
approach was not feasible without support external to ■ 
the^district. - ' ' . . . " 

^ A first and very important decision by anevaluator 

" is thi^ one: Are these funds, time, and\effort being . 
expended, to tielp, or. to change these students or these 
teachers or these administrator's? Or are long term . 
changes envisioned-changes yyhich will "1 iye long after 
the , project funding ends? If the stated goal of the 
hypotheiical project described above was a long term ' 
change, 'it must" be termed a failure. Jf the project 

» outline only covered the particulaif£tudent*and teacher 
population which existed at the^timepf the funding; it 
did not fail-efssummg each administrator, specialist ' n 
reading or testing, and 4 consultant' did, in tact, "defhis 
thing" welL Most* funded projects must l?e viewed as/ 
having some goals which are primarily of the long-term ^ 
variety. Tftese long-term goals may not be particularly * 
explicit in the proposal, but they are frequently 

• implicit in the relatively large amount of "money f spent 
oo*a small number of people. That is, When a district 
.syddenly begins to spend 50 percent Tiore per pupil on 
a certain group of students for a three year period- 

' then turnS the water off-it must be assumed they had 
something long-term, in mind. If not, the district 
probably wouJd have spread the money t eflually among 
. all of the pupils. ' \ ' ' i r v J 

How can you predict whether or notlthe project will 
.have a long-term impact? Some warning lights for 
. projects which probably will not have such ipnpact pan 
be offered. If a school district has* a curriculum*' 
building project* which does not clbse.ly "involve 'its 
regular curriculum people, -or a collie has' a project t 
1 wliich is staffed primaPLly^ by "outsiders" on "soft ' 
'.money", or if the," "regulars" 'in any agency are not 
closely involved .with the day to day operations of a 

* project, the possibility* of long term changes is clearly 
limited. ' - /' , . 
>* Tofind-'out ,if your# project is headed *toward the 
"short-term .oblivion" route, do. this littfe test. Peri- 
odically^ say one day every mcftith, make.a checklist of 
alllhe. things that happened in the proj'ect*dyring the 
day. Decide which) of these activities will continue to 
occur when the, funding runs. out. If most of the 

^*fifdtivtties Vrp dependent on outside moriey, the predic 




j£ on is ptjpty.clear. 



Evaluating Objectives; Two Interpretations . 

'Criterion-referenced tests, masjery learning, pe/- 
formance contracting,, learning packagesAbehavioral 
•objectives— these ^rrently aojSular tejjms are afl 
closely related to the notiopi T>f stating objectives 
specifically. The evaluator hafs the role pf "evaluating 
.the objectives/' People inte/pret this role in J wo, very 
different ways, and ft is Important that the project 
people and the evaluator are both "singing out of the 
same hymjjTbook" regarding the interpretation. 

To mOst/evaluators,' "evakjatipn of objectives" 
N means /'evaluating to see if the objectives have freen 
attaipeck" This implies measures wh^ich are most suited 
" to determining if an objective has been 'reached by the 
people toward whom the 'project was geared. The 
whole range, of techniques, mastery tests, question- 
naires, and interviews can be brought to bear on the 
objective. ' 

However, some people interpret "evaluating the" 
objective" to mean comparing the objective to other 
possfble objectives. That is, they see this as requiring a 
value judgment of the objective, compared to others. 
For example, take this objective: "The student shall 
recall the equivalencies between the common metric 
and- English units of time, length, volume and weight." 
To evaluate the attainmentof the objective, one would 
devise .some sor.t of an achievement test asking the 
student to recall all, or a ^random sample of, these 
equivalencies. 

- On the other hand, to place a value judgment on the 
objective would require asking questions like: Why 
.« should a student recall these equivalencies? Perhaps the 
\ student should simply recognize them, use thevn in 
context. Maybe estimating lengths; masses, and vol- 
umes is the proper manner Tn which students should 
' "know about" the relation between English and metric 
* .units. Even broader, why is this information important 
iat all? Valuable school time will j?e taken if the^tuSdent 
- is tq reach this objective Cotllcin't that time be better 
j Spent elsewhere, for. example, im reading a newspaper 
. or socializing with his peers? 

Evaluating to see rf the objectjye,has been attained is 
clearly the evaluator's-job, but placing value judgments 
, usually is not. These judgments should be made by the 
/people who are served by the agency "housing the 
. project. If the "housing agency" is a school/ then the 
people of the district .should make the value judg- 
^ ments. In cases where the situation at hand forces the 
evaluator to make these value judgments of objectives/ 
. I believe the evaluator should clearly delineete the two 
kinds in the final report. The' client has -the right fo 
21 know which evaluations are .technical judgments, and 
which are basically the evaluator's opinion. 



; / Concprriing*Car Salesmen: What Specifically 
did the Project Writers Promise to Do? 

"This Ml sweet s 'atts 'bout the bes' '67 in'fown.' Lo 
miles... 'bin treated like a balpy . . . bes' bargain in 
't<3m at four-fi-fty." 



The car salesman wraps one single statement of facJt 
in the same package with som& half-truths, im^lica- 
~» tions, and. fast talk. The only fact was the price. If the 
\ car was not actually the best '67 in town or if it is not 
the -best bargain in town, you won't really have any 
9 recourse later. The only, thing he actually said he 
' would c(o or guarantee was to sell the car for 
\ ''four-fifty.'-' 

'Proposal writers are o^ ; ten like car salesmen. The 
evaluator has to* separate the actual promises-the 
things the projectfwill do-from the other implications. 
Usually, the project staff will only be held accountable 
for things they specifically promised to do-just as the 
salesman is <only accountable for one fact in the 
Statement above. - - 0 
Now, mo^t proposal writers are not dishonest. The 
writer has to build-d* case # to. show' the background 
conditions which lead tcf the heed for additional funds. 
A competent proposal writer makes the -best possible 
case tor the proposed project. Sometimes it's-tiard to 
separate the "we will do this" statements from the 
"flag and motherhood" parts. Proposal writers are 
clever at mixing them up. It is possible to cut through 
the rhetoric, and go right to the heart of a proposal by 
looking immediately at a few key places. The two 
prjmary starting points are the budget and a project 
timetable, which is usually required in proposals. 

Use these two to set down a list of "will do" items.- 
If the budget lists money for a "field coordinator," 
then you can assume this is a "will dQ" item and not 
■ just rhetoric. The.timetable will probably tell you what 
this field coordinators supposed to be doing. I almost 
always begirt reading a new proposal at the' budget 
section, and then move to the activity section. Where 
' \% the project going to spend the money? The answer 
gives us a goad clue as to the real objectives. 

The budget has another important use. With it you 
can develop a hierarchy of objectives, from most to 
least important. Think • back to the hypothetical 
situation" Outlined previously < where the Assistant 
Superintendent tried 'to solve a reading problem. 
Suppose the budget looked like this; 



Administration 
Reading Specialists 
Test Specialist 
ln : Service Training 
Secretarial ' * 
Consultants • 



$ 20,000 
$ 84,000 
$ 10,000 
$' 1,000 
$ 6,000 
$ 5,000. 



Now^the evaluator prepares a score sheet: 



Administration 
Reading Specialists 
Test Specialist 
In-Service Traihftig 
Secretarial 
Consultants 



good 
X 

X 

X' • 



INDIFFERENT BAD 



That's 5 to 1-a good Qfoject. Right? Wrong! That's 
. 0' to 42, or 2 to bad- project. That is, $84,000 
spent on a section which didn't work out, while 

ERIC j 



$42,000 was spent on five sections which rateql 
"good," t v <; 

To develop the hierarchy of important objectives, 
try to allocate the money in th^ budget- to the 
different objectives. Some items, such as staff, travel 
and materials are easy to allocate, but others may have 
to be. skipped. After allocating, as many times as 
possible, make some sort of chart— a "pieces of pie" 

* circle graph, for example-to get an idea of the- relative 
importance of the objectives. The evaluation efforts 

'« should be similarly distributed. - • 

Timetable: If a Child is to be.Born in October, 
Conception Must Occur Somewhat Earlier , 

A proposal should contain a timetable of events. If 
it*does not, then the evaliJator should help the project 
develop one. If the one already developed is unrea- 
listic, then the eyaluatpr should help get it revised. 
Look at this objective: ./ #t 

"By June 1, 1975, the project staff will have** 
received the approval of the School Board to pilot te'st % 
a series of special reading programs- in three sch&ols in 
the district." 

Failure to plan ahead too often leads to'frantic and 
inefficient last minute etforts. What kind Qf^plapning- 
goes into the objective above? Start from the-.com- 
' pletion and work backwards— that usually is th£ easiest 
way. ' ft " 

The deadline i^i June 1st. What is the lastUchool 
Board meeting before June 1st? May 20th. And how 
long prior to the meeting'must members have material- 
which will'be acted upon? One month. (You're back to 
App\ 20th.) Hpvy'Jong to type and collate'thferepprt in 
the bureaucracy?^ wo weeks. (Now at April 6th.) How 
long to plah th^jprogram, including obtaining permis- 
sions, holding hearings, consulting "learned experts," 
and thing? li^e Miat? Three. mohths. We .are now at the 
first of the -Year. Must- staff be hiced? What other 
approvals are. needed arrdlrow long'wilj they take? The / 
Qress of day-to-day activities frequently can cause a 
project to avoid long range pfahnihg. The evaluator 
should help ke^p the project on schedule. 

The projeqt Staff: Concerning Prior t , 
, Commitments and Real Power I , , ' ' 

First fabte; jThe Assistant Superintendent wrote the 
proposal .qoyeVing the alleged reading and' diagnosing 
problem jn;hi^s district. The proposal,was funded. His- 
Superintendent thought it might be nice. for Mr. 
Assistant , Superintendent to direct the project, so the % . 
Schdol Bo^rid Ranted him a three-year'leaye of ^bsence 
from his V&gu^r job and narried, him. project director. 

Mr. Fontierj Assistant Superintending starts 7 work as t 
project director. &ot one^d^y, a few/vyeeksMater, the ' 
Superintendent is faped with, a problem that he knows 
the Formed Assistant Superintendent usecl to handle 
beautiful l^i So h'e asks'for'a small favor ./'just this one 
time." Tne|b favors probably ^A/ill continue, and soon 
the "proji9^|;director" , !Xworking far \e$& than full linra 
on tfie project. ! 




The moral of the fable: If someone onHhe project 
staff was with the same organization prior to appoint- 
ment to the project, look very carefully. Organizations 
—school districts, public agencies, universities, , etc/— 
.have a tendency to appoints person from withirrfor a 

, new job without appointing a different person td the 
old job. Since.the old job often goes unfilled, the new 
project person carries many of the old responsibilities 
with him. If a 'new Assistant Superintendent is not 
appointed to handle the responsibilities of/the Former 
Assistant, you can rest assured he is not committed full 
tfme to the project. . 

Final Fable: As project director 'the Assistant 
.Superintendent recruits two teachers from each build- 
ing to work with him on the project. The project^icks 

O P most' of the teachers'^ salaries. He directs these 



teachers to begin working on materials for six-* and « 
seven-year old children. One week later, he checks 
again with the teachers and finds two disturbing notes; 
First, they \]ad hot completed nearly as much as* he' 
expected; aad second, they were also developing- 
materials for pre-schoolers and eight-year olds? \r\ 
addition, they seemed resistive to his urgings. He found 
out, after some,, searching, that while on paper the 
teachers were paid by and responsible to the project, in 
practice anything that happened in a particular build- * 
ing was the responsibility of the building Principal. 
Even if the teachers had not been holapver teachers 
from prior years, the lines, 6f authority ^would have . 
been*blurred by thh traditions of the district^ 

The moral of this fable: Yoju can tell wm^has the 
"paper •power'' simply by reading the proposal. 5?e very 
sensitive' to the other issue, however, of "real" power. 
Who is always consulted when major decisions are 
•made? Who is able to.ACourrter or change directions 
given by the project personnel? i 

If' the lines of authority are not clear, the efforts of 



the project personnel 1 



be seriously diminished. If 



the teachers in the^/a^ovfe example do not know which 
person to respond/to/ (the project director or the 
building .principal), tney will probably not do a 
satisfactory job/Qf/eitner set of direction* BJu-rred lines 
of authority leaa ta all kinds of intrigue and ineffi- 
ciency^ The impfica^ton should not be drawn that the 
(project staff is ^ven absolute and unchallenged author- 
ity bver ' that/ which happens under the project's 
auspices. Ofpviously, these activities will affect the 
agency spqjpsoring'the project and the .agericy needs to 
in the decision-making. But "the project 
•abdicate all decisiqn-making responsiblity 
.the personnel in the sponsoring agency. A. 
u$t be struck and the project evaluator must 
nd make explicit (to atl, if possible,, but at 
the evaluation teaJn) just, what the specific 
'real" authority are. , . 





hsfve a 
staff .can 
in favor 
balance 
find qu, 
least 
lines, 

V 



the Evaluator, Too, Must Ask the Question, 
"Who Am I Working For?" 

etimes the evaluator is contacted by an agency 
to evaluate a project which was funded by the agency. 
The evaluiator is working for the, agency-helping the 
agency interact with the project, ^s in <the figure on the 
left. Sometimes, howgyer, the prbipeV- makes the 
contact with tjie evaluator-perhaps at the urging of 
the agency -THep the evaluator is Working with the/ 
project, helping £be project interact with the agency, as 
on the right. \ 



AGENCY PROJECT.. 

\ 

EVALUATOR 
Agency Hires Evaluator 



AGENCY - 



* PROJECT 



EVALUATOR 

# Project Hires Evaluator 



The evaluator must try to aCfewer the question, 
"Who amM working for?" which/usually leads to the 
questidn, "To what use ard myAresuJts. going to be' 



put?" The two questions beloog fbgether. So ver*y 
many times over the ppst few years I've had a long 
difficult discussion with project or agency people and 
finally put the question this Way: "Now look, folks, 
which (do you want: A hard-knocking internal evalua- 
tion which will tell you what's^ working— and what's 
hot— oV an evaluation which will accentuate the 'posi- 
tive' in an effort to sell the concept to outsiders? Is 
my report for you or. primarily for.outsjde consump- 
tion?" ' i*. 

Usually these people really \A/ant both-and that's 
possible. The evaluator can gather all of the proper 
information, letting the chips fall *whe re' they will, and f 
write two reports. One-is for internal consumption and 
the other for external public relations work. I'm not 
Suggesting that th£,second report be inaccurate in any 
yvay-only that it dwell more pn the positive notes, 
rather than pdinting otjt many ffaws in areas* which, 
need attention. 

Is that unethical? ^~>** 
• Clearly it's unethical if the "internal" report con- 
tains information about serious problems and the 
information is ignored. One part of tfti§ agreement 
must be that the project act on the suggestions giyen in 
the internal r v epor,t. If this is not done, then the 
evaluator should make the private feportpublic: 

But shouldn't this always be done — share all the 
results with everyone involved* — • project funding 
agency and public? In the best of all worlds the answer 
is clearly in the affirmative, especially when we see the 
devastating effects a "cover-up" can have, t However, a 
"cover-up" is not* what is involved here.- The true 
information would still go to the project.) But here is 

" the other side of the coin-and in the real world it's 
woFth considering: A project -or an agency can easily 
find a 41 "house ^valuator." A '"house evaluator" is 
someone who will figure out a way to get precisely t)ie 
results the project 'wants, to see. Now, if ten projects 1 
are vying for second year funding and nine* of them 
hire a N *'house evaluator," while the tenth g6ts a 
thorough evaluated, then the only one which will 
appear to have problems is. the tenth one. Unfortu- 
rjately, funding .agencies frequently do not look any - 

.deeper than the evaluation repprtT and this tenth 
project would not be refunded. When this is the 
situation—and" the' picture dra^ is not a hypothetical 

'«one, but vefy* real— then the idea of an internal and- 
external report is more defensible. „ - 

Fpr me, the most unhajBpy situations occur whef^ 
the project people want &nly an "accentuate the 

* positive"' report; but I wasn^ insigfitful enough to see 
this uhtrl all the work was done. A project I worked on 
had about $20,000 to.inerease ]the reading and math 
performance of 100,000 Jnner/city children, 'plus de- 
velop more positive affeGt within a four month span. , 
When the obvioCisVesuit occurred, I Reported it. Wy ' . 
role^as project evaluator .Was then terminated. This 
happened only one other time, and the unhappy part is 
knowing thqtyou've essentially wasted that time, since 

*,tf~" (9"~>fe are' not gding to implement any of your . 

eric ; : . • ' 
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.suggestions.' To avoid this type of situation, the 
eValuator should avoid people who are so jealous%or 
insecure of the project that they cannot accept 
anything but positive evaluations! 
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Continuous Assessment, 



m One very special domain of evaluation situations 
requires a special approach. These special situations 
involve schools. My" experience iq dealing with evalua- *. 
% tion problems brought by school 'administrators- 
excluding those associated -with projects funded by ^ 
oatside sources— is that theyfalj into three groups: 

(1) Evaluation of small, in-house projects to Kelp ^ 
decide whether or not they should continue. 'Sm3ll 

■ changes in curriculum, instructional delivery system, or*,, 
administrative organization of teachers and students 
. fall under. this heading. 

(2) 'Addressing a difficult question raised by some- 
one of some group to whom the administrator mus** 
attend. The question 'might come from a teacher or 
teacher group, a parent or parent group, a board 
member, or a newspaper'reporter. Two real situations' 
into which I've been drawn in the past'few months are 
these (cleverly reworded, I hope, to hide the actual 
cases): (, 

''Are the River Edge School's childretTTfalling behind 
the Creek Bank School's children in basic skills?" and 
"Has the ungraded plassroom solved last'year's scape- 
goring syndrome atJ/Vest School?" , . 

(3) Techniques for providing 'a systematic feedback 
"system to the public in a manner which will nbt be 
misunderstood. "Misunderstood" frequently means no 
more than cases, where'. the public applies absolute 
standards where relative interpretations ^re more ap- 
propriate. Like "80% mastery! Why, in* this world of 
numbers we need 1Q0%-yes* by God!-100%! When I m 
was a child .;.*". I 

If prior baseline data are not available— and such 
data almost never are— then tHese questions are diffi- 
cult* to answer^ JJnder such, conditions, the cost of* 
getting an evaluations nswer is simply unrealistic in the; 
face erf the conceivable benefits which could accrue 
therefrom. Needed is an inexpensive technique" for 
continuously accumulating baseline d^ta for all of the - 
major constituencies of the district. 

Stfcb a , program would have the following 
characteristics: 

• (a) Baselinf? data would .be^athered in all impor- 
tant areas qf the school's program. Included should be 
data on student achievement, affective measures'from 
current and former, students, affective fnform&tion 
from faculty, measures of community knowleflge, 
interest, and attitude, demographic information from 
the supporting area, and patterns, of student flow 
through the various mstructional^programs -qf the 
school. > 

^(b) * Continous^ data collection. ^Continuous" 
^viously does nglt mean daily, but means that data 
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will be gathered at fixed, pre set intervals. The internal 
' length should be a ^function ot the measure, ror 
example, demographic data change slowly, and a 
biannual interval would be satisfactory; whereas stu- 
dent achievement changes more quickly, and quarterly 
samples would be defensible. The faculty is fairly 
stable*, 1 and yearly samples seem appropriate. The size 
of the sample interval should be set with the district 
administrators to reflect their perceptions, of the 
change dynamics of each variable. 

(c) .Low cost feedback is an understandable format. 
If the system^ wou Id require outside funding for 
^ continued operation, or if the feedback would require 
h consultation \frith' measurement or^omputer experts 
every time, the administration seeksTto use it, theq the 
system will not have wi,de-rarrgincf applicability. The 
system should be initially established with a computer 
feedback system such that current district personnel 
can fedd the ijiost recent measures into the system to 
provide -updated reports ,on all of the measures 
•involved. Most districts already set aside some monies 
for ^valuation and research work, as well as having at 
least one administrative person deyoted to spending 
time' on these activities. Ohcie the continuous assess- 
ment system is, in operation, the* district should not 
have to invest substantially different amounts of ti 
and money as it had invested in the past. 

* (d) Tripd analysis. Educational data are, by an 
, large, • ordinal, at best. A statistic— a mean or a, grade 

equivalent— is usually more useful in the relative than 
in the absolute^nse. In making long-range decisions, 
or in deciding when td intervene in^ an existing 
program, trends are frequently more understandable 
than a table of figures. Computer generated graphs, 
with automatic statistical tests of significant-changes in 
the trendsr are an appropriate feedback system for each 
measure. Where* a* trend has, changed statistically, 

£ tabular data going back as far as possible would be 
provided, along with *a description of the meaning of 

" the observed change. The system should also note the 
interrelatedness of the measures and the. trends to 

• shows,the administration the points where events tdhd 
to happen together. _ t 

(e) Sampling is a key yyord for data collection. The* 
t primary data targgts involve groups. Individual mea- 
sures are not required fram qach person in the various 
groups. Takkig the" two key questions together-"How 
accurate must the results ±>e?" and "Haw much time' 
and money are available?"— the proper sample size can 
be determined. A very important phase of the igitial 
set-up of the project woul£ be* the generation of* 
specific random sampling techniques from the' dif- 
ferent populations. These directions would then be 
used by the district in" its systematic data collection 
' efforts. \ ' ' / N ... 
. Although f have sought a hospitable school district 
.for, the establishment of such a system for more than 
five years now, it is only recently /that a tentative- 
agreement to begiQ operations in f a district has been 
reachect, Aside frDm being a commentary on my 



ineptnessas a salesman, this difficulty offers a point on 
the "crisis orientation" of school administrators which 
is worth noting by evaluators. *This week's crisis has 
never happened before and may never happen again. 
Next week's or next month's crisis .isn't known yet. 
How can one be sure' the baseline data will help resolve, 
these* currentjy unforseen ^events? Additionally, a 
system based on trends cannot really begin functioning 
properly untiLthe measurements have gone through 
several cycles.. The idea of.investmg in a?system which 
cannot* be expected to pay off for a year or two is no^ 
attractive in the face of pressing current problems. 



Some Other Evaluator Roles 

Regardless of the size, funding, or location of the 
project, the 'evaluator has a unique opportunity to 
fulfill certain other less well-known roles. Three of 
these ar6 l^eepfng communication lines open, transla- 
ting numerical results into understandable terms for 
the project staff, and creating an "action now" 
philosophy about evaluation results* Let me expand 
briefly on each of these. 

Keeping the communication lines open. Every pro 
ject reaches a variety of groups. The groups usually 
have their own interests and have probably learned to 
communicate with each other in well-established ways. 
For example, the program with the Assistant Superin- 
tendent described earlier will involve students, tea- 
chers, administrators, Spjne reading .specialists, pro 
bably a few university professors, and maybe even 
some graduate students. Teachers and administrators 
.usually *have an employee-employer §ort .of relation 
ship. Two-way communication, even though badly 
.needed by the project, will be difficult to establish 
where prior tradition is strong. Building administrators 
£have established communication channels with central 
office administratorj;,the manners in which reading 
specialists communicate with university professors may 
be fixed, and so forth. To be successful, the project 
may require g lev?l of communication among groups 
which is not likely to occur without some sort of 
externally imposed greasing of the communication 
skids. * * 

The evaluator is, in a good position to be a 
communication facilitator. This is especially true if the 
evaluator is external-4hat is, not a regular or previous 
member of the agency housfng the project. If the 
evaluatdr is external to the agency funding the project, 
he will be relatively free from intimidatiorvor coercion 
from any group on the project or in the agency. From 
this perspective, it will be possible for him to establish 
communication ^between different groups by insuring 
anonymity to all*, who respond to his questions. 
Without anonymity, most people hesitate, to .make 
comments about'persdns higher up in the organization* 
especially if the comments might be construed as being 
critjcal. If the evaluator can insure anonymity and 
establish his own credibility, sthis hesitancy will eva 
^rn^r^ twp way communication will be possible. 
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Translating numerical results into practjcall meaning. 
The very last^part of the sentence above needs a few 
additional comments. The evaluation results aren't of 
any vatoe at all if the^people who make the decisions in 
the project cannot interpret them. Many ^project 
administrators are quite uncomfortable with numerical 
results of any kind. The "evaluator rfoust provide the 
project staff with more than just the results. Also 
provided must be information on probable implica 
tions, points of inaccuracy, information which may be 
unreliable or biased, and suggestions for further data • 
gathering. Evaluation reports which are' designed to 
help the project decision-makers should not be written 
as scholar^ articles for peers in the evaluation busi- 

♦ ness. They must be understandable by those who need 
the results. The test of the evaluation is not the 
technical beauty* of the measurement devices. Thfe test 
is the usefulness of ithe results. 

An action now philosophy. School administrators 
and project personnel-with notable exceptions, of 
course— view evaluators as not being within the main- 
stream' of the system or project. The evaluators work / 
is viewed as a necessary evil at best, and as.a threat at 
worst. In cases where the evaluator senses these kinds 
of attitudes, some public relations efforts are neces- 

v sary. These efforts should be directed toward convin- 
cing the staff that thorough evaluation efforts can 
enhance the probability that th£ objectives of the' 
project can be reached. The evaluator needs to 
, convince the staff that h6 or she is not, hiding behind 
-the curtains, waiting to expose .an embarrassing ipis- 
take. Evaluation, it should.be argued, involves 'arr 
"action now" orientation. That is, data gathered 
^throughout the project's life and fed to the staff very . 
quickly ,can be translated in^o early intervention in 
cases where things aren'* working as well as had been 
envisioned 1 .* The evaluator needs to convincingly 
demonstrate that evaluation can trapscend summative 
statements^ and that good formative evaluation can 
make positive contributions.*. 



A Final Word - 

This paper clearly does not constitute a "theory of 
evaluation." The comments are based on lessons 
learned i-n working with a diversity of evaluations 
projects. Funding on these .projects ranged from zero 
to millions of dollars. Not atl of the lesions were 
pleasant experiences. Unfortunately, more bitter les- 
sons ?re probably ahead. 

Is a definitive '"theory of evajuation" 'possible? 
Certain well-known . statements on evaluation have 
been written and they describe interesting and 
thought-provoking general evaluation approaches. But 
I have failed in efforts to operationqllze them. The 
theories are good ways to think about evaluation 
concepts, but too general to apply t€ specific projects. 
The projects are dimply too situation specific. 

Whether or f?§!t you agree witf\this philosophy, one 
current need is acutely apparent to me. This is the 
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need for a clearing house for evaluators *to describe 
unique approaches which have worked in diverse 
settings. The format should be closer to £opula 
Mechanics than Psychometnka. The present measure 
orient and research journals simply don't appear to be 
appropriate. m 

+ * 

When the Chicago £dard of Education officials, 
circulate an RFP for an evaluation project, the mailing 
list is around one hundred names. This list includes 
only some of those in the Chicago area who look upon 
therfiseh/es as "evaluators" for at least part of their 
profe&ional time. Across the country, the total list of 
people ^who have functioned as evaluators for schools 
*or projects must number in the thousands. There must 
be some very practical suggestions we can give one 
another. Somehow the professional^ measurement 



organizations should take a more active role in opening 
communication lines among these many evaluators. 

Evaluation is not research. Once the conditions have 
been established, the researcher' does not interact with 
with the experiment with thoughts toward changing 
conditions to insure significance. And evalt^tion is not 
•equivalent to measurement. ^Measurement applies to 
the devices used and their validation-the tests, inter 
views, performance tasks, or questionnaires used? 
•Evaluation is very much an interactive process where 
the technical expertise must interact with the political^ 
realities of the system, as well as the idiosyncratic 
personalities involved/ From my perspective, the pre- 
sence of an external evaluator has -a .very positive 
impact on both project and funding agency. In such a 
setting, the evaluator role is interesting, challenging, 
and worthwhile. ^ 
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